This disclosure relates generally to the field of image processing and, more particularly, to various blending techniques for use in generating wide area-of-view images constructed by stitching together individual image slices having varying exposures.
One conventional method to generate a wide area-of-view image from a sequence of images (frames) is illustrated in FIG. 1. To begin, a sequence of frames is captured (block 100), e.g., frames 1 through 3 shown in FIG. 1. The frames are then registered (block 105), identifying the regions of overlap between successive frames region 110 between frames 1 and 2, and region 115 between frames 2 and 3). As will be explained herein, in certain embodiments, it may be advantageous to only utilize and register the central regions of the images in the construction of a wide area-of-view image. These central regions may be referred to herein as image “slices” or image “slits.” In some preferred embodiments, the image slices may comprise the central one-eighth portion of the image frame, but any suitable slice size may be chosen for a given implementation. Once regions 110 and 115 have been specified, a path or seam is identified through each region (block 120). Here, seam 125 (through region 110) and seam 130 (through region 115) have been identified in FIG. 1. In accordance with standard scene-cut algorithms, seams 125 and 130 are generally selected to pass through the most similar pixels between each image found in their respective overlap region (i.e., frames 1 and 2 in region 110 and frames 2 and 3 in region 115). As a result, seams 125 and 130 are typically placed outside of, but immediately adjacent to, moving objects within the respective overlap region. With seams 125 and 130 identified, a blend operation across each is performed (block 135); the result of which is final wide area-of-view image 140.
The role of blending operation 135 is to mask or obfuscate the differences or transition between two frames. One approach to do this uses a process known as “gradient domain” blending, which consists of constructing the gradient field of final image 140 by copying the gradient fields of each image on the corresponding sides the identified seam (e.g., referring to identifier 145, the gradient fields across seam 125 would be gradient field A from frame 1 and gradient field B from frame 2). Once this is done, the final image is generated by integrating over the gradients across the seam. Reconstructing final wide angle-of-view image from its gradient fields requires substantial computational resources (e.g., to solve Poisson partial differential equations)—resources that may not permit for the satisfactory real-time generation of wide angle-of-view images on common hand-held devices such as, for example, personal electronic devices having embedded image sensors such as mobile telephones, personal music players, tablet computer systems, and personal gaming devices. Gradient domain blending may also be unable to mask registration errors visible along objects that span multiple frames.
Another approach, referred as “image cross-fading,” seeks to mask the transition between two frames by cross-fading pixel values from each frame along the transition seam (e.g. 125 and 130). This generally consists of a technique known as “alpha blending,” which comprises calculating a weighted average between the corresponding pixel values in the two frames, where the weight given to each pixel decreases smoothly while approaching the seam and vanishes at some distance after passing the seam. For example, the weight given to each pixel from frame 1 in region 110 can decrease smoothly from 1 to 0 while crossing seam 125 from left to right. Similarly, the weight given each pixel from frame 2 in region 110 can decrease smoothly from 1 to 0 while crossing seam 125 from right to left. Exactly on seam 125, pixels from both frame 1 and frame 2 may have the same weight, e.g., 0.5.
Reconstructing a final wide angle-of-view image using image cross-fading techniques alone can result in both “ghosting artifacts” (manifested by preserving parts of a moving object that is close to a transition seam) and banding artifacts (manifested in smooth areas in the images such as sky, constant color walls and fine textures). Likewise, using Poisson blending techniques alone can result in problems within regions of the reconstructing final wide angle-of-view image where, for instance, there are “broken” objects (e.g., a long object that is broken across individual image slices because of problems in the image registration process). Finally, in order to more fully capture image details in a wide angle-of-view image with varying brightness levels across the extent of the wide angle-of-view image, the auto exposure settings of the camera must be allowed to change or ‘drift’ during the capture process, preferably within some predefined bounds.
Thus, the inventors have realized new and non-obvious ways to constrain this auto exposure drift process and harness the information provided from the auto exposure drift process in order to more effectively account and correct for changes in the camera's exposure settings across consecutive image slices and blend across consecutive image slices without producing noticeable exposure banding artifacts, while still preserving maximum image detail.